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ABSTBACT 


In  the  early  phases  of  the  materiel  acquisition  procaaa, 

Subjaot  Matter  Exparta  (SMEa)  may  be  tha  only  aourca  for  data  on  a 
concaptual  waapon  ayatam.  In  thaaa  aarly  phaaaa,  tha  Early 
Comparability  Analyaia  (ECA)  Methodology  provldaa  a  tool  for 
identifying  tha  Manpower,  Paraonnal,  and  Training  (MPT)  raaourca 
intanaiva  taaka  (a.g. ,  tha  'high  drivers*)  on  currantly  fialdad 
ayatama  that  tha  concaptual  ayatam  will  raplaca.  Varloua  documanta 
ralatad  to  Manpowar  and  Paraonnal  Integration  (MANPRINT)  and  tha 
Concept  Formulation  Procaaa  (CFP)  diacuaa  riak  and  uncertainty.  By 
expanding  thaaa  concept a  and  uaing  aome  theoretical  conaidarationa , 
tha  ECA  method  can  ba  extended  to  provide  a  probabilistic  estimate 
of  tha  uncertainty  of  tha  SME’s  opinion  and  tha  risk  of  tha 
proposed  system.  Tha  procaaa  which  datarminaa  tha  risk  and 
uncertainty  probabilities  can  than  ba  worked  ‘backward*  to 
determine  tha  high  drivers  as  wall  as  tha  aourca  for  risk  and 
uncertainty.  Hypothetical  data  from  a  vary  small  sample  of  SMEa  is 
used  to  illustrate  tha  risk/uncartainty  procaaa.  Tha  method 
incorporates  a  consideration  for  ‘importance*  values  or  weights  in 
the  determination  of  riak/uncertainty  values  probabilities. 
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RISK  AND  UNCERTAINTY  IN  EARLY  COMPARABILITY  ANALYSIS 


1.  BACKGROUND : 

According  to  AR  002-2,  Manpower,  and  Personnel  Integration 
CMAMPR1NT)  in  the  Materiel  Acquisition  .Process,  the  purpose  of  the 
MANPRINT  philosophy  is  to  provide  a  systematic  approach  to  gather¬ 
ing  information  to  answer  the  question:  ‘can  this  soldier,  with 
this  training,  perform  these  tasks  to  these  standards  under  these 
conditions?*  In  the  early  phases  of  the  materiel  acquisition 
process,  there  may  be  a  lack  of  engineering  or  other  ’hard'  data  on 
the  proposed  system,  so  a  ‘comparability  analysis'  may  be 
required.  Data  are  usually  obtained  from  Subject  Matter  Experts 
(SMEs)  as  well  as  other  sources  (such  as  sample  data  collection  on 
the  predecessor  system  which  is  to  be  replaced).  Typically,  these 
front-end  analyses  which  are  performed  during  the  pre-concept 
phases  of  the  system  design  will  focus  on  the  predecessor  system 
and  on  lessons  learned. 

The  Early  Comparability  Analysis  (ECA)  methodology,  according 
to  the  ECA  Handbook  (Reference  b(2)),  is  a  ‘lessons  learned' 
approach  to  identify  the  Manpower,  Personnel,  and  Training  (MPT) 
resource  intensive  tasks  (‘high  drivers’)  on  currently  fielded 
materiel  systems  that  must  be  resolved  in  new  or  product  improved 
systems.  Thus,  a  major  focus  of  the  ECA  methodology  is  on  the 
'task.'  This  handbook  illustrates  a  procedure  for  identifying  high 
drivers  by  using  SMEs  and  other  data  and  some  easy  arithmetic 
calculations . 

The  draft  MANPRINT  Analysis  Methodology  (Reference  b(3))  shows 
that  ECA  is  a  part  of  the  pre-concept,  pre-milestone  zero,  phase  of 
MANPRINT  activities  and  is  one  of  several  processes  which  ‘feed* 
the  development  of  the  System  MANPRINT  Management  Plan  (SMMP) .  As 
issues  arise,  the  SMMP  documents  them.  As  the  issues  are 
addressed,  the  SMMP  is  updated,  thus  resulting  in  a  ‘living 
document'  which  changes  as  the  state  of  knowledge  changes  for  the 
proposed  new  system. 

To  get  an  indication  of  uncertainty,  we  look  at  the  Concept 
Formulation  Process  (CFP)  Memorandum  of  Instruction  (MOD  (See 
Reference  b(3)).  The  purpose  of  an  uncertainty  analysis  is  ‘to  put 
a  range  of  probabilities  around  the  estimate  at  a  specific  level  of 
confidence;  i.e.,  to  bound  the  estimate.*  Thus,  uncertainty 
involves  the  estimation  of  a  value  and  the  likely  or  probable  range 
of  values  that. can  occur.  When  we  define  a  critical  value  for  some 
system  performance  measurement,  any  value  which  exceeds  that 
critical  value  is  an  indication  of  an  unacceptable  condition. 

Thus,  if  there  is  a  range  of  likely  or  probable  values  which  exceed 
the  critical  value,  the  range  of  all  such  values  is  an  indication 
of  the  risk  probability. 


Th«  MANPBINT  Risk  Assessment  (MBA)  Quid*  (Reference  b(l)), 
contains  a  procedural  'tool  to  evaluate  MANFRINT  risk  associated 
with  the  development  of  an  emerging  materiel  system."  The  user  of 
this  guide  can  obtain  an  indication  of  the  'risk'  of  a  conceptual 
system  by  answering  a  series  of  Yes-No  questions.  As  in  the  MBA. 
the  risk  methodology  in  this  report  is  applicable  to  systems  in  the 
early  concept  phase.  Both  methods  can  be  used  to  identify  the  need 
for  further  analyses  or  voids  in  understanding.  However,  the 
methodology  in  this  report  will  also  provide  a  probability  estimate 
of  the  degree  of  risk  associated  with  the  system,  a  specific  task, 
and  the  components  of  each  task. 

We  will  look  at  uncertainty  and  risk  from  an  ECA  standpoint. 
From  the  previous  observations,  we  see  that  the  discussion  will 
focus  on  tasks  and  a  set  of  criteria  for  evaluating  task 
performance.  Risk  and  uncertainty  will  be  evaluated  with  respect 
to  estimated  values  of  task  performance  which  exceed  some  critical 
level  of  overall  acceptable  performance.  An  example  will  be  given 
using  judgements  from  five  SMEs ,  but  the  methodology  is  appropriate 
for  much  larger  samples  of  SMEs.  Indeed,  statistical  results  are 
much  more  reliable  when  larger  samples  (in  the  range  of  20  to  30 
SME)  are  used. 

Although  the  procedures  disaussed  herein  use  the  initial  SME 
results,  there  are  techniques  (such  as  the  Delphi  Method)  for 
allowing  the  ‘experts*  to  use  additional  information  to  'refine* 
their  opinion  (and  hence,  estimates).  The  caject  of  the  risk 
uncertainty  analyst*  should  be  to  use  all  of  the  information 
available  and  not  to  exclude  someone’s  opinion  simply  because  it  is 
not  the  same  as  other  estimates. 

2.  INTRODUCTION. 

We  will  use  the  data  in  Chapter  4  of  (draft)  TRADOC  Pam  602-1 
(Early  Comparability  Analysis  Procedural  Quide)  to  illustrate 
several  features  of  an  ECA  which  can  provide  additional  insight  to 
decision  makers.  Although  the  ECA  manual  strongly  suggests  a 
minimum  sample  of  ten  SME,  the  procedures  are  illustrated  below  and 
in  the  ECA  manual  using  only  five  SMEs.  First,  let  us  review  the 
ECA  methodology,  the  data,  and  how  it  was  collected  because  these 
are  important  considerations  in  the  validity  of  the  insights  which 
we  will  subsequently  address. 

The  ECA  methodology  depends  heavily  on  soliciting  information 
from  Subject  Matter  Experts  (SMEs) .  Most  often,  the  SMEs  will  be 
soldiers  in  the  Military  Occupational  Specialties  (MOS)  that  either 
operate  or  maintain  the  predecessor  system.  In  the  very  early 
phases  of  the  system  acquisition  process,  ECA  may  be  applied  to  the 
conceptual  system.  However,  SME  opinion  may  be  the  only  way  for 
collecting  data  for  an  ECA  on  a  conceptual  system  and  is  a  very 
appropriate  source  if  the  SMEs  are  truly  'experts.*  Step  3  of  the 
ECA  methodology  cal)s  for  compiling  a  list  of  tasks.  Step  4  calls 
for  collecting  data  from  the  SMEs  using  a  well-defined  structured 
approach  whereby  the  SMEs  rate  each  task  on  each  of  six  criteria. 
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For  an  ECA  for  which  there  ia  a  predeceaaor  ayatem,  a  myriad  of 
data  In  addition  to  SME  opinion  ahould  alao  bo  availablo.  Tho  aix 
crltoria  aro  ratod  on  a  varlablo  acalo,  with  poaaiblo  valuoa 
botwoon  ono  and  four.  A  acoro  of  ‘one*  uaually  ia  a  ‘doairablo* 
rating  (e.g.  ,  low  percentage  of  MOS  performing  tho  taak,  taak  not 
difficult,  taak  ia  aoldom  porformod,  low  taak  proficioncy  decay,  or 
email  number  of  houra  to  train)  and  a  acoro  of  *four*  ia  an 
‘undeair able*  rating  (e.g.,  high  percentage  performing  the  taak, 
taak  ia  difficult,  taak  frequently  performed,  high  proficiency 
decay,  or  large  number  of  houra  to  train).  Theae  * undeairable * 
oharaoteriatioa  may  reault  in  a  taak  that  ia  referred  to  a a  a  ‘high 
driver*  of  peraonnel  and  coat.  The  aix  taak  rating  criteria  are 
liated  in  Table  1,  and  the  aample  data  from  five  SMEa  for  the  aix 
criteria  are  liated  in  Table  2. 

Table  1.  ECA  Taak  Eating  Criteria 


Criteria 

Abbreviation 


Criteria 

Description 


PP 

TLD 

TPD 

FR 

DR 

TT 


Percent  Performing  Taak 
Taak  Learning  Difficulty 
Taak  Performance  Difficulty 
Frequency  Rate 
Decay  Rate 
Time  to  Train 


Table  2.  SME  Evaluationa  of  One  Taak 


SME 

1 

2 

3 

4 

5 


EE.  ILfi.  H2. 

4  1  1 

3  1  1 

3  2  2 

3  2  1 

2  1  2 


EE  £E  XX 

3  1  2 

3  1  3 

4  2  2 

4  3  1 

3  3  3 


Average 


3.0  1.4 


1.4  3.4  2.0  2.2 


The  ECA  methodology  reaulta  in  the  calculation  of  a  'Taak 
Score*  by  computing  the  product  of  the  averagea  of  the  SME  acorea. 
Uaing  the  data  above,  the  Taak  Score  ia: 

3.0  x  1.4  x  1.4  x  3.4  x  2.0  x  2.2  ■  87.06. 

Uaing  the  ECA  methodology,  thia  Taak  Score  ia  compared  to  an 
* eatabl iahed ’  cutoff  acore  of  216.  Since  the  Taak  Score  ia  leaa 
than  216,  the  taak  ia  not  conaidered  a  ‘High  Driver.*  The 
procedure  aummarized  above  would  be  completed  for  each  taak  to 
identify  the  ’High  Drivera*  of  the  ayatem  being  evaluated. 
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3.  UNCERTAINTY  AND  RISK. 


Whereas  the  ECA  mathod  provides  useful  information  on  currently 
fielded  systems,  as  we  will  see  below,  with  modifications,  it  can 
provide  useful  information  on  conceptual  systems  as  well.  The 
modifications  discussed  below  will  enhance  the  basic  methodology  for 
evaluating  either  conceptual  or  currently  fielded  systems.  First, 
note  in  Table  2,  the  SMEs  have  some  disagreement  in  their 
assessments  of  eaoh  of  the  six  criteria.  The  ECA  methodology  seeks 
to  form  a  concensus  by  using  the  average  value  for  each  of  the  six 
criteria.  However,  the  variation  of  the  SME  evaluations  is  an 
indication  of  their  uncertainty.  We  could  also  argue  that  high 
variation  in  SME  opinion  may  be  an  indication  of  the  likelihood  that 
problems  will  be  encountered  in  the  concept  formulation  phase  of  the 
system  acquisition  process.  Certainly,  variation  in  SME  opinion 
translates  into  uncertainty. 

To  illustrate  the  need  for  further  ECA  methodology  enhance¬ 
ments,  suppose  that  we  could  get  another  ‘set’  of  five  SME 
responses.  On  the  basis  of  the  first  sample  of  five,  we  should 
conclude  that  the  second  sample  would  probably  also  exhibit  some 
uncertainty.  Of  course,  to  gain  'perfect'  information  on  the 
uncertainty  of  the  SMEs  (e.g.,  to  determine  their  variation  in  task 
criteria  scores)  ,  we  would  have  to  gather  every  SME’s  opinion. 
However,  if  we  are  willing  to  consider  the  first  sample  as 
‘representative*  of  any  and  all  samples  of  SME  opinion,  we  can  use 
probability  and  statistics  to  make  inferences  about  the  other 
possible  samples  of  SME  opinion. 

If  we  carry  the  considerations  on  the  SME  sample  further,  with 
some  assumptions  about  the  relationship  between  the  six  criteria,  we 
can  use  the  SME  uncertainty  to  provide  us  information  on  the  'risk* 
of  the  task  as  well.  First,  let  us  assume  that  the  SMEs  evaluated 
each  of  the  six  criteria  separately  (that  is,  for  example,  criterion 
one  was  not  influenced  by  nor  did  it  influence  the  rating  given  to 
criterion  two) .  We  refer  to  this  property  as  independence.  This 
assumption  was  implicitly  built  into  the  ECA  methodology  when  the 
Task  Score  was  computed  using  the  product  of  the  average  criteria 
values.  Although  a  desirable  property,  as  discussed  in  the  Post 
Script  (Section  9)  ,  independence  is  not  necessary  for  the  ECA 
methodology . 

We  can  aonsider  the  SME  criteria  scores  to  be  taken  from  a 
distribution  of  scores.  As  we  noted  earlier,  we  cannot  practically 
collect  all  SME  scores  to  gain  'perfect'  information,  so  we  will  use 
this  sample  of  SMEs  to  give  us  an  idea  of  the  distribution 
characteristics .  That  is,  we  can  use  the  sample  characteristics  to 
estimate  how  the  entire  SME  population  would  have  scored  each 
independent  criterion.  First,  wo  will  construct  a  distribution  of 
possible  criterion  scores  using  the  SME  data  to  estimate  key  para¬ 
meters  which  describe  the  distributions .  Next,  we  will  construct  a 
distribution  of  Task  Scores  by  randomly  drawing  values  from  the 
sample  criterion  distributions .  The  procedure  for  constructing  the 
Task  Score  distribution  will  be  illustrated  later.  We  will  now 
discuss  the  characteristics  of  the  criterion  distributions . 
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There  &r«  several  theoretically  sound  distributions  to  use. 

The  'easiest*  distribution  would  be  a  normal  distribution  whose 
average  and  variance  (or  standard  deviation)  is  estimated  by  the 
sample  data  average  and  variance  (or  standard  deviation).  However, 
we  have  to  place  some  constraint  on  the  distribution  o f  possible 
values.  In  a  normal  distribution,  it  is  possible  (but  not  very 
probable)  to  have  a  score  several  standard  deviations  below  the 
average  (or  mean).  This  could  equate  to  a  meaningless  score  of  '0' 
or  a  negative  criterion  value.  3ince  the  sample  of  SMEs  could  not 
provide  any  score  other  than  a  value  between  1  and  4,  we  must 
‘truncate*  the  theoretical  normal  distribution  so  that  only  values 
between  1  and  4  are  possible.  Of  course,  we  could  open  up  the 
range  of  allowable  SME  scores,  thereby  permitting  the  SMEs  to 
provide  more  precise  evaluations. 

Another  distribution  that  we  could  use  is  referred  to  as  the 
‘triangle*  distribution,  which  is  characterized  by  the  pessimistic 
(low),  most  likely  (average),  and  optimistic  (high)  scores.  In  the 
triangle  distribution,  these  three  parameters  are  readily 
determined  using  SME  sample  data  and  the  distribution  requires  no 
‘truncation.*  The  implications  on  the  choice  of  either  the  normal 
or  triangle  distribution  will  be  discussed  later. 

Other  types  of  distributions  may  also  be  considered  in  the 
risk/uncertainty  analysis,  and  the  selection  of  the  type  of 
distribution  certainly  should  not  be  based  on  ‘the  easiest  to  use* 
criterion.  To  help  make  an  appropriate  selection,  the  SME  data 
should  be  plotted  to  see  if  it  resembles  the  proposed 
distribution.  SME  responses  need  not  be  normally  distributed  (or 
exhibit  any  particular  type  of  distribution)  to  conduct  a  risk/ 
uncertainty  analysis.  Use  of  normal  and/or  triangle  distribution 
herein  is  a  simple  matter  of  convenience. 

Given  our  acceptance  of  the  six  criteria  and  the  ability  to  use 
SME  sample  data  to  construct  a  distribution  of  all  possible  SME 
scores  (using  the  SME  sample  to  estimate  the  population 
parameters) ,  we  can  now  construct  a  theoretical  population  Task 
Score  by  randomly  sampling  from  each  of  the  six  distributions  and 
calculating  the  product  of  the  randomly  sampled  values.  The 
process  of  constructing  the  distribution  of  Task  Scores  is 
illustrated  in  Figure  1. 

We  note  in  Figure  1  that  there  is  a  ‘counter’ ,  which  is 
'iterated'  or  increased  as  we  calculate  Task  Scores.  Thus,  each 
particular  count  is  frequently  referred  to  as  an  ‘iteration.’  To 
construct  a  ‘good*  distribution  of  Task  Scores,  several  thousand 
iterations  wild  normally  be  required,  thus  necessitating  a  computer 
to  automate  the  process.  If  we  denote  each  randomly  selected 
(triangle  or  normal)  criterion  variable  as  Xt  (i  »  1,2, 3, 4, 5,  or 
0)  ,  then  the  J-th  Task  Score  value  is  computed  as: 

Task  Scorej  *  XiXaXaX^XaX*  (the  product  of  criterion  scores)  . 


Not*  i 
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W«  are  now  prepared  to  aaaeaa  tha  uncertainty  aaaociatad  with 
tha  taak  uaing  tha  procedure  illuatratad  in  Figure  1.  Firat,  wa 
computa  tha  Maan  Product  Taak  Scora,  which  la  aimpiy  tha  product  of 
tha  maana  of  tha  aix  critarion  diatr ibutiona  (wa  know  tha  maana 
ainca  we  calculated  them  to  daacriba  tha  aampla  diatribution 
paramatara  for  both  tha  normal  or  triangle  diatributiona) .  Tha 
Uncartainty  area  liaa  to  tha  right  of  tha  Maan  Product  Taak  Score 
aa  ahown  in  Figure  2. 


Uncertainty 


Figure  2.  Riak/Uncartainty  Uaing  Normal  Criteria  Diatribution 


Tha  Uncartainty  ia  moat  often  axpraaaad  in  terma  of  a 
probability.  Wa  can  eatimate  tha  uncartainty  probability  by 
counting  tha  number  of  timaa  tha  Taak  Score  axcaada  tha  Maan 
Product  Taak  Scora.  (Recall  that  tha  ECA  methodology  computed  tha 
product  of  tha  tnaana  of  tha  aix  critarion  acoraa  to  compare  to  tha 
cutoff  acora.)  Tha  probability  ia  than  calculated  by  dividing  thia 
count  by  tha  number  of  itarationa  or  'counter'  value.  For  tha 
reaulta  ahown  in  Figure  2,  there  ware  434  timaa  that  the  Taak 
Scoraa  axcaadad  tha  Maan  Product  (87.805)  out  of  tha  1,000 
itarationa,  ao  tha  uncartainty  probability  for  thia  taak  example  ia 
0.434. 
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The  uncertainty  probability  tails  ua  that  there  is  leaa  than  an 
even  chance  (or  probability  leaa  than  50  percent)  that  some  other 
sample  of  SMEs  could  have  given  ua  a  Task  Score  greater  than  the 
Mean  Product  Task  Score.  The  uncertainty  probability  for  the  Task 
Saore  distribution  is  a  composite  of  the  uncertainty  (disagreement) 
in  the  six  criterion  distributions.  The  value  of  uncertainty, 
then,  is  entirely  a  result  of  this  disagreement  relative  to  an 
average  composite  (Mean  Product  Task  Saore) . 

As  we  observed  earlier,  the  ECA  methodology  compares  the  Task 
Score  to  an  established  cutoff  score  of  210.  Mow,  however,  we  have 
a  Task  Score  distribution  to  compare  to  the  established  cutoff 
score.  The  risk  in  the  task  is  computed  as  a  probability  using  a 
procedure  similar  to  that  which  calculated  the  uncertainty 
probability.  That  is,  we  count  each  time  that  a  Task  Score  value 
exceeds  the  established  cutoff  value.  The  risk  probability  is 
calculated  by  dividing  that  count  by  the  number  of  iterations.  For 
the  example  in  Pigure  2,  the  risk  probability  is  0.079,  a 
moderately  low  value  (some  would  say  that  a  risk  of  5  percent  or 
more  is  'significant'). 

What  the  risk  probability  is  telling  us  is  that  despite  a  large 
uncertainty  (probability  ■  0.434),  the  likelihood  that  the  SMEs  (or 
another  sample  of  SMEs)  would  exceed  the  established  cutoff  value 
is  only  70  times  out  of  1,000  (probability  »  0.079). 

At  this  point,  we  must  somewhat  disassociate  the  uncertain  ty 
from  risk.  In  another  situation,  our  SMEs  could  have  absolutely 
agreed  on  a  rating  for  a  task  (uncertainty  equals  zero  percent) , 
but  the  Task  Saore  aould  have  exceeded  the  critical  value  (risk 
equals  100  percent).  Also,  as  we  will  see  later,  it  is  entirely 
possible  for  the  risk  probability  to  exceed  the  uncertainty 
probability.  Thus,  uncertainty  does  not  necessarily  cause  or  lead 
to  risk. 

So  far,  using  this  methodology,  we  can  provide  management  with 
not  only  the  ’standard*  ECA  methodology  information,  but  we  can 
also  quantify  the  risk  and  uncertainty  as  well.  As  we  will  see, 
however,  we  aan  provide  even  more  information  to  a  decision  maker. 

Perhaps  the  easiest  enhancement  to  make  (with  the  intention  to 
provide  more  Information)  is  to  change  the  established  cutoff 
scores  to  'critical*  values  which  are  'tailored*  to  the  situation 
being  evaluated.  From  Figure  2,  it  can  be  seen  that  raising  or 
lowering  the  critical  value  will  affect  the  value  of  the  risk.  The 
value  216  (the  ECA  methodology  established  cutoff  value)  is 
arbitrary  and  may  be  considered  to  be  too  low  or  too  high, 
depending  on  the  situation  being  evaluated.  Also,  we  compute  a 
particular  Task  Score  by  forming  the  product  of  individual 
criterion  scores .  By  working  'backwards,*  we  can  conclude  that 
each  criterion  has  a  cutoff  imputed  which  is  the  sixth-root  of  216, 
or  roughly  2.45  (that  is,  2.45-  to  the  sixth  power  is  about  216). 

This  means  that  we  have  established  that  any  score  for  any 
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criterion  above  2.40  la  a  ’risk*  indicator.  Aa  wa  aamplad  from  tha 
alx  crltarla  and  formad  our  dlatrlbutlon  of  Taak  Scoraa ,  we  addad 
up  tha  counta  of  thoaa  Taak  Scoraa  which  conalatantly  axcaadad  tha 
critical  valua,  tharaby  giving  ua  a  ‘risk*  aaaaaamant. 

4.  IMPORTANCE  AMD  WEIGHTS. 

At  thla  point,  wa  might  maka  an  obaarvatlon  that  wa  may  not 
conaldar  aaoh  crltarlon  to  hava  aqual  lmportanca  In  datarmlnlng 
rlak.  Tha  lmportanca  laaua  can  ba  axpllcltly  addraaaad  by 
’walghtlng’  tha  crltarla  dlffarantly.  Aaauma  for  tha  momant  that 
wa  obtalnad  tha  following  ‘lmportanca*  walghta  from  our  panal  of 
SMEa  ualng  a  procadura  which  wa  will  axplaln  balow: 

Tabla  3:  Exampla  of  Importanaa  Walghta. 

Normal lzad 

Dlatrlbutlon  Balatlva  lmportanca  Walghta 


PP 

10 

1.50 

TLD 

5 

0.75 

TPD 

1 

0. 15 

FR 

18 

2.70 

DR 

2 

0.30 

TT 

-t 

9.,-gP 

TOTAL 

40 

e.oo 

In  tha  aolumn  labeled 

’Relative  Importance* 

in  Tabla  3,  FR 

not  only  tha  moat  important  crltarlon,  it  la  18  timaa  tha 
importanaa  of  TPD ,  aa  datarminad  by  our  SME  aampla  ualng  tha 
procadura  which  will  ba  diacuaaad  balow.  Thaaa  valuaa  of  'Balatlva 
lmportanca*  muat  ba  tranaformad  into  walghta,  whara  tha  valuaa  of 
tha  walghta  muat  hava  cartain  propartiaa. 

First ,  alnca  wa  hava  alx  crltarla,  tha  aum  of  tha  walghta  muat 
aqual  alx.  Thla  la  naadad  to  anabla  ua  to  compare  ‘weighted* 
raaulta  to  'unweighted*  raaulta.  (Actually,  unweighted  raaulta 
hava  an  Imputed  relative  importance  for  each  criteria  of  1,  a o  tha 
aum  of  tha  relative  importance  aquala  tha  number  of  criteria) .  Wa 
can  tranaform  or  'normalize*  tha  relative  walghta  by  dividing  each 
of  tha  relative  walghta  by  their  total  and  multiplying  by  tha  valua 
0.  Tha  normalized  weights  are  in  tha  right  column  of  Tabla  3. 

(For  exampla,  tha  weight  for  PP  is  found  by  dividing  tha  Relative 
Importance  by  40  and  multiplying  tha  result  by  6) .  Wa  sea  that  tha 
ratio  of  tha  weights  is  tha  same  aa  tha  ratio  of  tha  relative 
importance.  For  exampla,  tha  PP  to  TPD  ratio  ia  10/1  for  tha 
relative  importance  and  1.50  to  0.15  for  tha  weight.  Tha  ratio  of 
importance  has  bean  preserved  so  that  tha  1.50  weight  is  still  10 
timaa  tha  0.15  TPD  weight. 

To  perform  a  r iak/uncertainty  analysis  using  weighted 
(importance)  criteria  and  Taak  Scoraa,  tha  calculation  procadura  ia 
changed  somewhat  aa  follows: 
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Task  Score  •  Xiw“  x  X*w'*  x  ...  x  Xew%#, 

Risk  Value  -  CVi"*1  x  CVaw*a  x  ...  CV«W%*.  and 

Mean  Product  Teat  Score  ■  X i w% 1  x  Xaw%a  x  ...  X«w%*. 

Whara  Xi  is  a  randomly  salactad  scora  from  tha  i-th  criterion 
distribution  (i  ■  1,  2,  ...0  crltaria) , 

CVj  is  tha  critical  value  (constant)  for  tha  j-th 
criterion,  and 

Xj  is  tha  mean  of  tha  J-th  criterion  distribution. 

Thus,  tha  weights  are  exponents  for  tha  variables  which  are  used 
for  determining  tha  Task  Scora  distribution,  critical  value,  and 
Mean  Product  Task  Scora.  Whan  each  weight  (wtt)  has  a  value  of 
1.0,  tha  results  are  identical  to  tha  previous  discussion  results 
using  'unweighted*  values. 

There  are  several  similar  techniques  which  use  the  geometric 
mean  for  calculating  weights.  Care  should  be  taken,  however,  to 
assure  that  the  'normalizing'  process  results  in  weights  whose  sum 
equals  the  number  of  criterion.  Also,  other  techniques  impose  a 


Risk 


Figure  3.  Weighted  Uncertainty/Risk  Using  Normal  Criteria 
Distributions 
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range  of  allowable  eoorea  (e.g.,  a  fixed  reaponee  scale)  and  require 
pair-wise  comparisons  of  every  criterion  to  every  other  criterion. 
Obviously,  if  a  task  is  evaluated  on  a  large  number  of  criterion, 
the  number  of  pair-wise  comparisons  could  become  large.  See 
References  c(3)  and  c(4)  in  Section  8. 

Using  the  weighting  procedure  and  the  weights  in  Table  3,  we 
obtain  the  Distribution  of  Task  Scores  shown  in  Figure  3. 

The  only  change  in  the  method  used  to  generate  Figure  3  and 
Figure  2  is  that  Figure  2  had  equal  criteria  importance  (weights) 
and  Figure  3  used  unequal  importance  (weights  in  Table  3).  Note, 
however,  there  is  a  somewhat  modest  increase  in  Uncertainty,  but  a 
very  large  risk  probability  of  75.2  percent  I  Admittedly,  the 
importance  values  were  selected  to  give  such  a  dramatic  change  in 
risk  to  illustrate  a  point.  From  Table  3,  PP  and  FR  had  the  largest 
weights  and  from  Table  2,  these  also  had  the  highest  averages.  This 
means  that  the  contribution  to  the  Task  Score  distribution  due  to 
these  two  criteria  was  far  more  than  the  contribution  due  to  the 
other  four  criteria.  Obviously,  then,  high  criterion  scores 
translate  into  high  Task  Scores,  which  translated  into  high  risk. 

The  point  being  illustrated  is  that  risk  is  influenced  by  the 
importance  placed  on  the  criteria  and  is  not  influenced  by  the 
uncertainty  of  the  SMEs . 

Using  Table  4,  we  can  illustrate  one  method  for  calculating 
importance  weights. 

Table  4.  Relative  Weight  Example. 


CRITERION 


SME 

IL 

Xb& 

TPD  » 

E£ 

£& 

H 

1 

0.0 

4.0 

1.0 

11.0 

3.0 

7.0 

2 

10.0 

7.0 

1.0 

20.0 

1 . 5 

2.5 

3 

10.0 

5.0 

1.0 

25.0 

0.5 

3.0 

4 

8.0 

4.0 

1.0 

18.0 

3.0 

3.0 

3 

JLSL 

0.0 

U 2. 

19.9. 

3_juJ2l 

0.5 

Geometric 

10.0 

5.0 

1.0 

18.0 

2.0 

4 . 0 

Mean  ** 

*  Arbitrarily  set  to  1.0  as  'reference' 
**  Rounded  to  a  whole  number 


For  PP ,  the  Geometric  Mean  ■  PPx  x  PPa  x  PP*  x..PP« 

A  similar  calculation  is  used  for  other  5  criterion  geometric 

means . 

In  Table  4,  SME  1  felt  that  PP  was  nine  times  as  important  as 
(relative  to)  criterion  TPD,  which  was  given  a  reference  value  of 
1.0.  Thus,  each  SME  rates  each  criterion  importance  relative  to  the 
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reference  criterion,  ao  the  value*  ara  ralativa  lmportanca  (palativa 
to  tha  ' raf aranca ' ) .  Wa  nota  hara ,  aa  in  tha  SME  avaluationa  in 
Table  2,  that  tha  SME  have  aoma  uncertainty  in  their  aaaaaamant  of 
ralativa  importance.  Tha  Geometric  Mean  ia  aimply  the  5-th  root  of 
tha  produot  of  tha  SME  acoraa  for  each  criterion.  Tha  5-th  root  ia 
uaad  in  thia  oaaa,  ainca  wa  have  five  SME*.  If  wa  have  ‘n*  SMEa ,  wa 
would  taka  tha  n-th  root  of  tha  product  of  tha  SME  ralativa 
importance  value*  for  each  of  the  six  criterion.  It  may  occur  that 
aeveral  SMEa  would  conaiatantly  rat*  a  criterion  lower  than  1.0  (the 
’reference").  Wa  can  see  that  SME  3  rated  DR  at  0.5  (meaning 
"lower*  or  1/2  aa  Important  aa  tha  reference).  If  tha  SMEa 
conaiatantly  rata  a  criterion  at  lea*  than  the  "reference, ‘  the 
Geometric  Mean  will  alao  be  lower  than  1.0.  W*  aaw  in  Table  3  how 
tha  Ralativa  Weight*  (Geometric  Mean*)  ar*  converted  into 
"normalized*  weight*.  We  ahould  alao  obaerve  that  no  negative  or 
zero  value*  are  allowed  aa  SME  relative  weight*  ainca  auch  an 
evaluation  haa  no  practical  meaning.  (If  the  SME  feela  that  the 
importance  of  a  criterion  ia  very  amall,  a  value  auch  aa  10“* 
could  be  given,  but  not  zero.  Anv  zero  value  will  make  the 
geometric  mean  equal  to  zero  regardleaa  of  al 1  of  the  other  acoraa. 
The  "dampening"  effect  cauaed  by  on*  SME  providing  very  amall 
numbera  decreaaea  rapidly  aa  the  number  of  SMEa  (e.g.,  the  aample 
aize)  increaaea  to  the  20-30  rang*. 

Aa  with  the  calculation  of  the  Taak  Score*  (which  ia  aimilar  to 
the  Geometric  Mean  in  that  it  involve*  taking  the  product  of 
numbera) ,  the  ua#  of  the  geometric  mean  for  calculating  the 
importance  weight*  require*  aeveral  assumption*  to  be  met  and,  a a 
will  be  shown,  "builds*  certain  properties  into  the  values  thua 
calculated. 

First,  w*  must  assume  that  the  SMEs  rank  the  relative  importance 
of  each  criterion  independently  of  each  other  and  that  the  ranking 
of  on*  criterion  does  not  affect  (or  ia  not  influenced  by)  another 
criterion  (other  than  the  "reference*  criterion).  Secondly,  w*  must 
also  assume  that  the  values  given  by  the  SMEs  ar*  samples  of  all 
possible  SME  values.  With  these  two  assumptions,  w*  can  us*  the 
properties  of  the  SME  relative  importance  distribution  to  infer 
something  about  the  importance  of  the  criterion  if  we  could  get  all 
SMEa  (vs.  a  sample  only)  to  rat*  the  criterion. 

However,  unlike  our  sampling  procedures  used  to  build  the  Task 
Score  Distribution,  we  need  a  set  of  specific  weights  to  us*.  Yet, 
we  want  certain  statistical  properties  to  be  present  when  we 
calculate  these  weights.  The  two  desired  statistical  properties  a re 
that  the  procedure  should  capture  both  the  overall  (concenaus)  value 
and  be  sensitive  to  wide  variations  in  SME  rating*  (e.g., 
uncertainty  or  disagreement) .  Thus,  the  selection  of  the  techniques 
for  calculating  the  importance  weights  is  not  arbitrary:  it  must 
possess  certain  properties.  The  data  in  Table  5  will  be  used  to 
illustrate  the  effect  of  disagreement  of  SMEs  on  several  commonly 
used  atatiatica. 
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Table  5.  Illustration  of  Statistical  Properties 


A1 ternatives 


.... .  SME 

1 

2 

3 

1 

4.30 

3.00 

7.00 

2 

5.00 

3.60 

1.00 

3 

3.00 

4.60 

7.00 

4 

2.00 

3.00 

4.00 

9 

4.60 

1.00 

Mean 

3.50 

3.80 

4.00 

Variance 

1.20 

0.31 

7.20 

Geometric  Mean 

3.32 

3.73 

2.87 

Illustrated  in  Table  5  are  the  scores  by  five  SMEs  on  three 
alternatives.  The  particular  interpretation  of  the  meaning  of  the 
alternative  or  score  is  not  important.  If  we  were  to  select  an 
alternative  with  the  highest  mean  (average) ,  we  would  select 
Alternative  3,  with  a  mean  of  4.0.  Note,  however,  that  Alternative 
3  had  the  most  ‘disagreement*  among  the  SMEs  as  evidenced  by  the 
highest  variance.  On  the  other  hand,  Alternative  2  has  a  somewhat 
lower  mean  (3.80)  and  has  the  lowest  variance  (hence,  the  most 
agreement  by  the  SMEs) . 

The  geometric  mean  has  the  property  that  the  highest  value  is 
achieved  when  the  variance  is  the  lowest  when  the  mean  remains 
constant.  For  example,  try  any  three  positive  (not  zero)  numbers 
that  add  to  some  value.  Change  the  numbers  at  will  so  long  as  their 
sum  remains  the  same.  We  will  find  that  the  geometric  mean  of  the 
three  numbers  will  always  be  less  than  or  equal  to  the  arithmetic 
mean  (the  average)  and  will  equal  the  arithmetic  mean  only  when  all 
three  numbers  are  identical  (e.g.,  the  variance  is  zero). 

The  implication  of  this  is  that  the  geometric  mean  combines  some 
properties  of  both  an  ordinary  average  and  a  variance.  Using  this 
technique  for  calculating  the  relative  weights  ‘builds'  in  several 
desirable  statistical  properties.  First,  if  one  relative  weight  is 
greater  than  another  (is  more  Important) ,  it  must  have  consistently 
higher  scores  and  the  SMEs  must  agrea  that  that  it  is  an  important 
criterion  (the  variance  is  low).  Conversely,  a  particular  criterion 
may  have  a  low  weight  (importance)  either  because  all  of  the  SMEs 
uniformly  rated  it  low  or  the  criterion  had  a  large  variance  in 
score  (the  SMEs  disagreed  on  its  importance) . 

9.  TASK  CRITERIA  DISTRIBUTIONS. 

Thus  far,  we  have  not  discussed  the  implications  of  the  choice 
between  a  normal  or  triangle  distribution  for  the  six  criteria. 
Figures  2  and  3  respectively  showed  the  unweighted  and  weighted 
distribution  of  Task  Scores  using  a  normal  distribution  for  all  six 
criteria  (the  distributions  differ,  however,  in  their  mean  and 
variance,  but  look  alike  in  that  they  are  normally  distributed).  In 
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th«  unweighted  oaae,  we  found  a  email  but  eignif leant  plek 
probability  of  0.078.  Piguree  4  and  5  ehow  the  unweighted  and 

TMk  ??0PV’ ,P**p,otiv*ly '  ueing  all  triangle  diatributione 
for  the  aix  criteria  (each  haa  the  loweat  SMS  value,  the  average, 
and  the  hlgheet  SMS  value  ae  ita  paramet era) . 


Uncertainty 


Figure  4.  Unweighted  Taak  Scorea  Ueing  Triangle  Diatributione 
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From  Figure  4,  we  can  see  that  tha  riak  probability  dacraaaad 
substantial ly  (0.4  parcant) ,  whila  tha  uncartainty  probability 
inoraaaad  lonawhat  (51.3  parcant)  ralativa  to  tha  rasulta  in  Figure 
2.  In  this  situation,  this  is  to  ba  expected  sinca  tha  triangla 
distribution  ‘cuts  off'  soma  of  tha  values  which  tha  normal 
distribution  allows  (avan  though  tha  normal  distribution  is 
‘ truncatad ' ) .  For  axampla,  with  a  triangla  distribution  with  low  * 
3.0,  madium  ■  3.4,  and  high  ■  4.0,  thara  will  ba  no  critarion  scoras 
ganaratad  lowar  than  3.0.  However,  a  normal  distribution  with  an 
avaraga  aqual  to  tha  triangla  distribution  * s  avaraga  (3.4)  can  hava 
valuas  lowar  than  1  or  highar  than  4  wara  it  not  for  tha  ‘trunca¬ 
tion’  which  wa  imposad  (nota,  truncation  takas  placa  bacausa  wa 
imposed  a  limit  of  a  scora  from  or.s  to  four  on  tha  SME’s  choica,  and 
tharafora,  wa  imposa  this  on  allcwabla  critarion  scoras  as  wall). 

Tha  triangla  distribution  with  a  ‘high*  (optimistic)  paramatar 
of  2.0  can  ganarata  valuas  no  highar  than  2.0.  Howavar ,  for  tha 
sama  critarion  with  a  normal  distribution  with  maan  of  1.4  and 
standard  daviation  of  0.55,  random  numbars  up  to  two  standard 
daviations  should  ba  ganaratad  (valua  of  2.5).  Thus,  tha  triangla 
distribution  ganaratas  lowar  critarion  valuas,  maaning  it  givas  a 
lowar  astimata  of  risk  in  this  case.  This  changa  may  not  always  ba 
prasant  whan  *raal*  data  and  situations  ara  usad  (for  axampla,  if  a 
triangla  distribution  has  a  low  valua  of  1  and  a  high  valua  of  4,  it 
will  provida  valuas  as  high  and  low  as  a  normal  distribution  with  a 
maan  of  2.5  and  varianca  of  1.0.  Howavar ,  tha  frequency  of  high  and 
low  valuas  will  diffar  batwaan  tha  normal  and  triangla  distribution. 

Nota,  howavar,  that  tha  risk  in  Figura  5  has  incraasad  from  tha 
risk  in  Figura  3  avan  though  tha  only  changa  is  in  tha  distribution 
typas  for  tha  critarion  variablas.  Wharaa ■  Figura  3  usad  tha  sama 
waights  usad  in  Figura  5,  tha  critarion  distributions  changad  from 
normal  to  triangla  and  tha  risk  incraasad  baaausa  of  tha  waighting 
and  tha  typa  of  distribution  of  tha  critarion  variablas. 

Tha  practical  implication  of  this  discussion  is  that  the  choica 
of  tha  scales  for  SMEs  to  rata  is  critical  (in  this  discussion,  tha 
allowable  range  was  1  to  4  to  reflect  tha  ECA  methodology).  Whan 
tha  range  of  tha  scale  is  incraasad,  wa  will  most  likaly  saa  more 
variation  in  responses,  but  tha  r isk/uncartainty  methodology  can 
accommodate  this  much  batter  than  tha  ECM  methodology.  In  fact,  to 
gat  a  batter  r isk/uncartainty  assessment,  tha  range  of  allowable 
scores  for  SMEs  should  be  widened  to  tha  maximum  extant  possible. 

«•  OVERALL  RISK. 

Thus  far,  wa  hava  explicitly  addressed  tha  r isk/uncartainty  of  a 
single  task.  Assume,  for  purposes  of  illustration,  that  tha 
proposed  system  has  tan  tasks  that  ara  independent  (that  is  tha 
completion  of  one  task  is  not  dependent  on  and  does  not  affect  tha 
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completion  of  another  task) .  By  applying  the  risk/uncertainty 
analysis  as  previously  discussed,  we  can  obtain  the  risk/uncertainty 
of  each  individual  task.  Through  probability  theory,  as  shown 
below,  we  can  determine  the  overall  risk  for  the  proposed  system. 
This,  of  course,  implies  that  we  could  select  one  alternative 
configuration  over  another  by  computing  the  alternative  risks  and 
selecting  the  lowest  risk  alternative.  Additionally,  if  the  overall 
risk  is  too  high,  we  can  ‘backtrack*  through  the  analysis  to  find 
the  high  risk/uncertainty  tasks/criteria  and  single  these  areas  out 
for  further  study.  Study,  would  then  be  used  to  reduce  the 
uncertainty/risk  by  ‘getting  a  handle*  on  what  the  source  of  the 
high  risk  is. 

Table  0  illustrates  the  risk  assessments  of  six  independent 
tasks  for  a  proposed  system.  We  should  note  that  these  are 
individual  tasks .  and  are  not  the  criteria  used  to  evaluate  a  task 
as  shown  in  Table  2.  We  can  calculate  the  overall  risk  by  taking 
the  product  of  the  ‘unrisk’  (that  is,  one  minus  the  value  of  the 
risk)  and  subtracting  the  result  from  1  as  shown  below: 


Table  6.  Multitask  Risk  Assessment 

Task  Risk  1-Rlsk 

1  0.20  0.80 

2  0.10  0.00 

3  0.01  0.90 

4  0.30  0.70 

3  0.01  0.00 

0  0.33  0.65 


Overall  Risk  -  1-C(.80)  x  (.90)  x  (.00)  x  (.70)  x  (.00)  x  (.65)] 

■  0.670. 

Overall,  there  is  a  high  risk  for  this  system  (probability  * 
0.679).  By  ‘backtracking,*  we  can  see  that  there  are  three  tasks 
(1,  4,  and  6)  that  are  significant  factors  in  the  overall  risk 
determination.  Further  ‘backtracking’  can  be  done  to  point  to 
specific  risk  areas  (the  criterion)  and  provide  management  with  a 
means  of  selectively  using  further  study  and  analysis  to  reduce 
risk/uncertainty  and  to  identify  Issues  needing  more  intensive 
investigations . 

If  the  tasks  are  not  independent,  we  should  not  use  the  product 
form  of  overall  risk.  A  computer  model  of  the  analysis  could  be 
developed  to  assess  the  risk  if  the  type  of  task  dependency  is 
known.  Also,  some  tasks  may  be  more  important  than  others  (for 
example,  some  task  may  cost  more  than  others),  so  an  overall  risk 
determination  can  be  made  using  task  weights  and  a  methodology 
similar  to  the  weighting  of  criterion  distributions .  There  would  be 
a  variety  of  methods  which  could  be  used  (including  SME  opinion)  to 
determine  appropriate  task  weights. 
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7.  CONCLUSIONS. 


The  uae  o f  techniquea  which  reduce  a  aet  of  data  to  only  an 
average  do  not  papmit  ua  to  uaa  all  of  tha  information  whioh  might 
ba  obtainad  from  our  Subjact  Mattar  Experta  (SMEa) .  By  axplicitly 
oonaidaring  tha  SME  diaagraamant ,  wa  can  datarmina  uncertainty,  and 
by  comparing  tha  calculated  Taak  Scoraa  to  an  appropriate  critical 
value,  wa  can  oalaulate  tha  riak  in  propoaad  material  aoquiaition 
ayatema . 

Thia  uncertainty  and  riak  analyaia  can  ba  uaad  to  guide  tha 
naada  for  further  analyaia  to  reduce  tha  riak  that  critical  taaka 
will  beaome  ’high  drivera'  in  tha  aoquiaition  procaaa.  SMEa  can 
alao  provide  valuable  information  on  tha  importance  of  individual 
criteria  uaad  to  evaluate  tha  propoaad  ayatam,  thereby  directly 
influencing  tha  determination  of  tha  riak  aa  wall. 

Wa  have  aaan  that  a  high  uncertainty  doaa  not  naaaaaarily  imply 
a  high  riak.  Uncertainty  ha a  to  do  with  tha  amount  of  diaagraamant 
of  our  axparta  and  riak  haa  to  do  with  the  likelihood  that  aoma 
critical  conaidaration  will  exceed  aoma  predetermined  value. 

Wa  have  alao  obaarvad  that  tha  choice  of  acalaa  that  tha  SMEa 
uaa  in  evaluating  tha  criteria  will  definitely  affect  tha  riak/ 
uncertainty  determination  ainca  valuea  will  ba  'truncated*  whan  wa 
aampla  from  tha  SME  criteria  diatr ibutiona .  Accordingly,  whan 
performing  a  riak/uncertainty  ECA,  tha  range  of  allowable  SME 
raaponaaa  ahould  ba  widened  aa  far  aa  poaaible  ao  aa  not  to  raatrict 
unnaturally  tha  choice  of  SME  acoraa  (thereby  dacraaaing  our  ability 
to  aaaaaa  their  diaagraamant  and  hence,  uncertainty) . 
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A  POST  SCRIPT. 

The  objective  of  this  paper  is  to  be  provocative.  Admittedly, 
there  are  many  considerations  which  have  not  been  addressed.  The 
purpose  of  the  paper  is  to  illustrate  that  existing  methodology  can 
be  expanded  when  proper  consideration  is  given  to  the  analytical 
methodologies . 
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When  the  first  drafts  of  ths  papsr  wsr*  distributed  for 
comments,  several  discussion  points  emerged.  Although  the 
discussion  below  may  not  address  all  such  possible  considerations 
(the  literature  abounds  in  such  considerations) ,  it  does  address  and 
illustrate  the  need  for  further  work  in  the  ECA  methodology.  The 
discussion  below  concludes  with  a  recommendation  on  the  development 
of  an  Expert  System  for  use  by  non-technical  people.  Such  a  system 
would  have  built  in  features  to  appropriately  incorporate  the 
theoretical  considerations  discussed  below. 

Independence  of  criterion  scores  loomed  as  a  major  concern  to 
several  readers.  If  we  were  using  the  criterion  scores  to  build  an 
index  or  some  other  scale  which  could  be  used  to  estimate  or  predict 
a  proposed  weapon  system's  performance,  we  would  want  them  to  be 
independent  in  a  statistical  sense  as  well  as  independently 
evaluated  by  our  SMIs .  In  the  use  of  ECA  methodology,  there  is 
evidence  to  suggest  that  Task  Learning  Difficulty  (TLD)  and 
Time-to-train  (TT) ,  for  example,  are  not  statistical ly  independent. 
Other  criteria  (or  subscores)  may  also  be  correlated.  Therefore, 
when  we  multiply  the  six  criterion  variables  to  obtain  the  ECA 
score,  one  might  argue  that  the  score  is  biased  because  of  the 
correlation  between  the  subsaores.  For  the  ECA  methodology  however, 
the  independence  consideration  is  not  an  issue. 

We  are  interested  in  relative  measurements .  Unlike  an  IQ,  for 
example,  which  might  be  estimated  by  a  series  of  subscores  on  a 
‘standardized*  test,  the  ECA  score  has  no  intrinsic  value  in  and  of 
itself.  The  ECA  score  takes  on  importance  only  with  respect  to  some 
arbitrarily  established  cutoff  point  which  is  a  composite  of  the 
individual  subsaores  themselves.  Any  ‘bias*  in  the  ECA  score  will 
be  similarly  present  in  the  cutoff  score. 

If  we  Insist  on  concerning  ourselves  with  correlation  between 
subscores  or  criterion  variables,  we  might  look  at  several  possible 
causes  of  correlation.  To  a  large  extent,  the  reason  for  dependency 
rests  in  the  use  of  a  very  restricted  range  of  SME  allowable 
scores.  Certainly,  widening  the  allowable  range  of  scores  (e.g., 
from  a  1  to  4  range  to  a  range  of  1  to  7)  gives  the  SME  more 
choices.  If  SMEs  use  the  increased  range,  we  should  see  the 
‘dependency*  reduced.  Suppose,  for  example,  that  five  SMEs  rate  a 
criterion  which  has  a  range  of  1-4  for  allowable  scores.  The  first 
four  can  each  select  a  different  score,  but  the  fifth  must  select  a 
score  which  duplicates  at  least  one  of  the  previous  four  SME 
saores .  Thus,  the  SME  scores  will  become  statistically  dependent 
simply  because  we  restricted  the  allowable  range  of  saores.  If  we 
expand  the  range,  we  increase  the  allowable  responses.  Thus,  an 
objective  of  Increasing  the  allowable  range  is  to  increase  the 
variance  in  the  SME  response  (assuming,  of  course,  that  the  SME  will 
use  the  increased  range).  However,  the  wider  allowable  range  of 
saores  may  increase  the  SMEs’  difficulty  in  discerning  more  subtle 
diflerenoes  in  the  ratings. 
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II,  nonetheless,  a  reasonable  goal  is  to  increase  variance, 
another  variance-increasing  technique  actually  is  present  in  the  ECA 
methodology.  The  product  form  of  the  ECA  creates  the  maximum 
variance  in  the  ECA  score  where  there  is  little  variance  in  the 
individual  subscores.  Recall  that  the  risk/uncertainty  method  is 
essentially  concerned  with  variance.  Varianoe  in  SME  opinion 
translates  into  uncertainty.  Variance  in  ECA  scores  that  are  above 
a  certain  ‘critical*  value  is  a  measurement  of  risk.  Variance  is 
caused  by  disagreement  of  the  SME.  Thus,  we  want  a  method  which 
calculates  the  most  variance  from  our  sample  of  SMEs . 

Note  that  a  method  which  calculates  an  average  score  not  only 
does  not  generate  the  maximum  varianoe,  it  also  allows  for 
‘compensatory*  ratings.  That  is,  if  one  SME  scores  one  point  above 
the  average  and  another  scores  one  point  below  the  average,  for 
example,  then  the  two  SME  have  compensated  for  each  other,  thereby 
‘averaging  out*  their  disagreement.  The  product  form  of  the  ECA 
allows  for  no  such  averaging  out  or  compensation  of  SME  scores  and, 
in  fact,  allows  us  to  focus  on  disagreement. 

Another  cause  for  apparent  dependency  may  be  that  the  criterion 
variables  are,  indeed,  related.  For  example,  if  a  task  is  difficult 
for  the  soldiers  to  learn,  it  probably  is  also  difficult  to  train, 
and  difficult  training  tasks  most  likely  will  take  more  time  to 
train.  Hence,  TLD  and  TT  might  reasonably  be  expected  to  be  related 
or  dependent. 

If,  however,  we  wish  to  construct  an  ECA  score  which  accounts 
lor  such  dependency,  we  can  simply  weight  the  individual  subscores 
by  some  appropriate  value.  For  example,  if  TLD  and  TT  were  the  only 
two  correlated  subscores,  we  could  give  each  one  a  weight  of 
one-half.  Another  technique  for  handling  the  correlated  variables 
is  to  perform  a  factor  analysis  on  the  criterion  variables  to 
determine  the  minimum  number  of  independent  factors  and  the  equation 
which  transforms  linear  combinations  of  subscores  into  factor 
saores.  The  r isk/uncertainty  method  can  then  be  applied  to  the 
factor  scores.  We  could  also  expand  the  number  of  subscores  and  use 
one  of  several  available  techniques  (discussed  above)  to  adjust  for 
dependency  if  we  feel  this  is  necessary. 

The  point  being  made  is  that  the  lack  of  independence  in  our 
subscores  is  not  a  sufficient  reason  to  discount  the  usefulness  of 
the  ECA  methodology  nor  should  it  even  necessarily  be  a  concern.  We 
are  not  necessarily  after  an  absolute  measurement  scale.  If  we  are 
concerned  about  correlated  subscores,  techniques  exist  to  ameliorate 
perceived  difficulty. 

This  paper  obviously  did  not  address  a  number  of  other  issues 
concerning  the  use  of  the  ECA  methodology.  For  example,  the  issue 
of  inter-rater  reliability  should  be  addressed  in  the  adequacy  of 
the  design  of  the  data  collection  Instrument. 
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This  paper  alao  did  not  deal  with  how  the  raaulta  could  ba 
iaplamantad.  Tha  analyaia  portion  of  tha  ECA  riak/unoartainty 
methodology  haa  bean  automated.  We  propoaa  that  an  Expert  Syatem 
for  ECA  be  developed  aa  an  Artificial  Intelligence  application.  The 
ayatem  would  include  a  processor  which  would  allow  the  uaer  to 
'tailor*  the  questions  for  SME  to  rate  and  would  have  various 
consistency  checks  built  in  (i.e.,  internal  rater  consistency, 
inter-rater  reliability,  etc.).  Obviously,  such  a  capability  will 
take  more  effort  to  complete,  so  this  paper  should  be  viewed  as  only 
a  start  in  the  process. 


FOREWORD 


The  Early  Comparability  Analyaia  (ECA)  methodology  was 
davalopad  by  tha  US  Army  Paraonnal  Intagratlon  Command  (USAPIC)  and 
was  approved  for  uaa  aa  a  Manpower  Paraonnal  Integration  (MANPRINT) 
mathodology  in  February,  19B5.  It  waa  daaignad  aa  a  laaaona 
learned  approach  for  analyzing  thoae  taaka  performed  by  aoldiera  on 
currently  fielded  ayatema.  In  addition,  it  waa  daaignad  to  be 
eaaily  performed  by  TRADOC  action  officera  uaing  reiourcea  that  are 
readily  available.  The  identification  of  'high  driver'  taaka, 
thoae  taaka  which  aignlf icantly  affect  Manpower,  Paraonnal,  and 
Training  (MPT)  reaourcea,  is  completed  by  a  relatively  aimple 
aoreening  mechaniam.  The  labor  intenaive  work  ia  to  conduct  a  taak 
analyaia  on  the  high  driver  taaka. 

Many  potential  enhancementa  to  the  methodology,  according  to 
USAPIC,  were  deliberately  omitted  in  the  deaign  of  the  methodology 
even  though  they  may  have  been  aomewhat  beneficial.  The  overall 
deaign  goal  waa  to  keep  the  methodology  eaay  to  perform. 

With  modi f iatlon ,  the  baaic  ECA  methodology  can  alao  be  applied 
to  conceptual  ayatema  early  in  the  materiel  acquiaition  proceaa. 
Thia  report  demonatratea  how  riak  and  uncertainty  meaaurement 
techniquea  can  be  applied  to  the  baaic  ECA  methodology  to  permit  an 
early  MPT  aaaeaament  for  a  conceptual  ayatem.  In  addition,  the 
techniquea  can  be  applied  to  fielded  ayatema.  The  riak  and 
uncertainty  procedurea  provide  a  meana  of  making  the  ECA  effort  to 
identify  high  drivera  more  rigoroua  if  the  reaourcea  to  do  ao  are 
available . 


